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ABSTRACT 

This paper presents findings of an exploratory study 
undertaken at Marist Sisters' College in Woolwich, Sydney (Australia) 
to examine student use of multimedia software packages. Students 
(n=70) in years 7~12 were randomly selected to complete an evaluation 
of a popular multimedia package, "Encarta," on the following factors: 
ease of use, navigation, cognitive load, mapping, screen design, 
knowledge space complexity, information presentation, media 
integration, and overall functionality. Findings included: (1) 46% of 

students commented on the limited quantity of information; (2) 12% of 
students expressed difficulty managing the search process; (3) 65% of 
students felt that the electronic information search was easier and 
more efficient than a traditional print search; and (4) 11% indicated 
that the task was more fun, interesting and exciting than when using 
print sources. The major disadvantage of using electronic over print 
resources was identified as the lack of depth and complexity of 
information in the search package. Another major disadvantage, not 
related to the Encarta program, was computer system downtime. The 
study discusses the implications of these findings for essential 
effective information literacy instruction in electronic information 
environments, as well as curriculum directions for schools. (Contains 
11 references.) (SWC) 
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This paper presents the findings of an exploratory study undertaken at Manst Sisters College m 
Woolwich, Sydney to examine student use of multimedia packages. The study provides a 
student-centered evaluation of a popular multimedia package "Encarta", identifies problems with 
the use of this package, and discusses the implications of these fmdmgs for effective information 
literacy instruction in electronic information environments, as well as curriculum directions for 
schools. 

Theoretical Framework 

Multimedia, defined generally as the use of several media such as text, graphics, 
animation, audio and video integrated in a program that is delivered on a stand-alone computer 
workstation or via a computer network, is regarded as one of the most important factors shaping 
schooling for the 21st century. As an information resource, it is full of gloss with its vivid graphics 
and images, creative screen dissolves, state-of-the-art animations, millions of colors and limitless 
variety of sights and sounds. It's novel, it's difference and it has a certain magnetic appeal. You 
can confirm this by walking off the street into any "Timezone" games parlors. In a short period of 
time it has captured the attention of school students across the globe. (Insert stats. On growth of 
CD Roms). Current predictions of growth rates in the development of multimedia packages 
suggest that publishing of multimedia will overtake the publication of books by the turn of the 
century, and that such sources will be an important aspect of collection development for school 

lib rar ies 

There is potential for multimedia to make a huge impact on curriculum design, teaching 
processes, learning strategies and tasks, information seeking and the structure of the learning 
environment itself. However, the key question remains: will multimedia fulfill its promise? 

Critical analysis and evaluation of multimedia lags well behind the development of new packages. 
Much of the evaluative literature on multimedia tends to focus on media and technical 
While these might be important qualities, this study recognizes the immediate need ol the 
profession to elucidate and test evaluative criteria that focus on pedagogic dimensions and student 
usability. Multimedia can be aptly described as continuously work-in-progress. This can only be 

effectively done through careful formative and summative evaluation processes. 

As both teachers and information professionals, teacher- librarians need to understand 
how multimedia impacts on learning. This is made more critical given the hype that accompanies 
many multimedia packages. Take a moment to read any of the unevidenced claims embedded m 
the publishers' blurbs that are being used to entice educators to purchase such products. In one 
recent catalogue, we found the following claims: 

"strengthens their deductive reasoning skills" 

"increased productivity" . „ 

"develops critical thinking skills, active reading and a high level of comprehension 

"motivates independent reading" 

"encourages students to use cooperative strategies" 

"turns your students into real research writers" 

"teaches students to work together" 

To date, there is very little data to substantiate any of these claims. 

Underpinning this study is the premise that decisions informed by thoughtful evaluation 
are better than those based on just guesswork, ignorance, publishers' propaganda or superficial 
perusal. Evaluations of technology-based packages in the past have tended to adopt simplistic 
approaches to evaluation, focusing primarily on achievement of goals. However, recent literature 
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reflects a greater concern with merit, worth, or value, user perceptions and key concerns and 
issues Kazlauskas (1994) for instance identifies a number of learner characteristics that should 
be considered when designing instruction. These include: cognitive characteristics such as 

aptitude, developmental level, language development, reading level visual literacy cognitive 
processing style, prior knowledge, learning strategies and general world knowledge, physiological 
characteristics such as age, sensory perception and health (fatigue); and Psychology 
characteristics such as interests and attitudes, motivation, experience with media anxiety, locus 
of control, peer relationship, socio-economic background, and affihataon aspects, ^ese ^clw- 
acteristics of learners that define the utility of any instructional product, including multimedia. 
Inherent in this idea is the promise that understanding user characteristics and user learning 
needs should form the framework for making judgements about any instructional package m any 

mediumprom & gimilar pergp ective, Reeves (1992) and Reeves and Harmon (1993) have identified 
a range of user-centered dimensions that could form the focus of evaluation of multimedia 
packages. They are in two broad areas: user-orientated dimensions, and pedagogic dimensions 

(Todd, 1995). 

User-Oriented Dimensions , 

Ease of Use. This refers to how easily the learner interacts with the multimedia package, 

and is an aggregate of many of the dimensions that follow. 

Navigation. This refers to the perceived ability to move through the contents m a deliberate, 

Cognitive D I^ad r What is the mental effort of learning with multimedia? It is important 
that the structure of the package is clear to learners so that they can efficiently manage 
simultaneously all the choices and tasks demanded of multimedia, and not be contused y 

numerous options that increase cognitive load. , , , 

Mapping. This refers to program’s ability to track and represent the learner s path throug 

the program, to avoid the problem of user disorientation. 

Screen Design. Does screen design violate principles of screen design or follow the 
principles? In particular, there should be careful thought given to the appropriateness ol 

iSowlSd^Sp^ce Complexity. This focuses on the network of concepts and relationships 
that compose the mental schema a learner possesses about a topic. It raises the question ol 
expert knowledge versus novice knowledge. It is important that the domain knowledge is 
structured in a way that learners can create an appropriate semantic organization relevant to 
their learning tasks and can make inferences about their state of knowledge m relation to an 

"expert" model of knowledge to be learned. . , . , 

Information Presentation. This refers to whether the information contained is presented 
in an understandable form. The most elegantly designed interface is useless if the 
information is incomprehensible to the learner. It is important that they can comprehend 
analyze, synthesize this information in ways appropriate to their learning tasks, and so that 

links can be made to existing knowledge. 

Media Integration. How much is it truly a multimedia program and how well does it 
combine the different media to produce an effective whole? Do they work together to form a 
coherent program or is it a hodge podge of gratuitous media segments? , , 

Aesthetics. Is there an overall artistry in the production and design? Does the product 
possess a beauty or elegance that goes beyond novelty effect? Are the different media used for 

OveralTFunctionality This relates to perceived utility of the program. While multimedia 
packages can have multipleuses, does its overall functionality meet its stated objective; does 
it meet the specific intended use that currently exists in the mind of the learner? Does it 
enrich the curriculum are for which it is intended? 
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Pedagogic Dimensions 

Teachers tend to make the intuitive assumption that each media type makes a unique 
contribution to learning. There is the additive assumption that instruction presented in two 



mediums produce more learning that instruction in one medium, and the multiplicative 
assumption that instruction integrating a range of mediums such as using multimedia packages 
produce even more learning (Clark and Craig, 1992). Within a strong educational framework the 
challenge of multimedia lies in using applications that actively engage the learner. Interfaces that 
are proactive rather than merely an exercise in page turning, pressing buttons to present yet 
another stream of information, reject role-learning and empower learning "by doing" and reflection. 
It is this notion of empowerment that underpins the idea of interactively. Other dimensions 
identified by Reeves (1992). include: 



1 Epistemology. This relates to theory of knowledge held by designers. Is the package 
objectivist, that is, concerned with accurate information and where learning is seen as 
acquiring truth. Or does the package advocate a constructivist, epistemology, reflecting many 
viewpoints regarding a topic and providing a full range of options from which learners can 

construct their own knowledge. . 

2. Pedagogical Philosophy. Does it merely transmit information, such that the learner is, at 
best a button pusher, or does it provide an active learning environment that addresses 
unique interest, styles, motivators, capabilities of individual learners? 

3. Underlying Psychology. Does the package function primarily to shape behaviors through 
stimulus, response, feedback and reinforcement, or does it provide a wide variety of learning 
strategies that foster the construction of meaning and understanding through opportunities 
for analysis and synthesis of information? This relates to the notion of interactively. 

4 Goal Orientation. Does the package enable students to set sharply focused goals? 

5. Instructional Sequencing. Are the problem solving activities purposefully designed to be 
intrinsically interesting and challenging, enabling learners to build coherent and logical ideas 
in new meaningful and relevant contexts? 

6. Experiential Validity. To what extent does the package provide opportunities for learning 

situated in real world experience? ... . , r 

7. Role of Instructor. Is the package "teacher-proof' - is it merely an authontanan provider ol 
knowledge, or is it designed to enable teachers to have critical roles as students interact with 
the package? 

8 Value of Errors. Does the package provide opportunities for learning from mistakes, or are 
the potential responses arranged in such a way that learners can only make correct 



9. Motivation. Is the learning context intrinsically motivating, that is, motivation that is 
integral to the learning environment, or extrinsic, coming from outside the learning 
environment? 

10. Structure. Structure can vary from tightly prescribed pathways to widely divergent options. 
Although low levels of structure may seem to promote increased individualization, learners 
may become confused and lose track of what they are doing. 

11. Accommodation of Individual Differences. How does the package accommodate 
differences in aptitude, prerequisite knowledge, motivation, experience, learning style. What 
cognitive scaffolding is provided to support learning? e.g. advanced organizers, outlines, 

content maps, time estimates. . 

12. Cooperative Learning. How does the package facilitate instructional methods m which 

learners work together to accomplish shared goals. 



The Evaluation Study , 

This exploratory study was conducted at Manst Sisters College, Woolwich, Sydney m early 
1995. The college is a Catholic, systemic, secondary girls school with 700 students enrolled from 
Year 7 to 12. The college s information infrastructure combines the traditional, predominantly 
paper-based library resources and networked information technology. The technological 
component consists of an electronic library catalogue, a curriculum applications server and 14 
CD-ROM drives networked to 57 workstations throughout the college. The information technology 
provides users with access to the College’s library catalogue, to desktop publishing, 
teaching/leaming packages, internet and multimedia from workstands situated m a computer 
laboratory, classrooms, staffroom, library and information laboratory adjacent to the library 
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facility. "Encarta" the Microsoft multimedia package was evaluated in the study. 

"Encarta 94" as described by Microsoft starts with the complete text of the 29-volume 
Funk and Wagnal's New Encyclopedia plus many new articles, photos, ammations, detailed 
illustrations, music segments, and sounds. According to the publisher, this package harnesses 
all the power of the personal computer to inspire curiosity, open the door to wonder, and take your 
family on a learning journey that never ends. It makes learning engaging and easy by Pitting a 
world y of knowledge right at our fingertips". In this study students were asked to evaluate their 

experience of the package without any of the publisher's type. , , , , , 

Seventy students were involved in this evaluation study. Students were selected randomly 
over one week as they worked on the multimedia workstations around the school. Students were 
initially asked if they has used "Enact" and only those students with previous experience (that is 
they had used "Encarta" before to access information) were included in the study. A prolile ot 
these students is provided in Table 1. 



Class Level 


Number 


Percent 


Year 7 


17 


24 


Year 8 


01 


01 


Year 9 


28 


40 


Year 11 


06 


09 


Year 12 


18 


26 



Table 1 



Seventeen percent of the students, primarily in Year 7, indicated that they were novice 
users, having used "Encarta" only a few times previously. 77% of the sample described 
themselves as average users, that is, they were quite comfortable with using the package and had 
used it at least on several occasions previously. They were drawn mainly from Years 9, 11 an 
12. 6% of the sample described themselves as expert users, primarily from years 11 and 12, who 
had used the package on many occasions, and felt very comfortable with searching through l 



Data Collection „ . 

A questionnaire was developed to indicate the evaluative reactions to Encarta in terms 

of: Ease of Use, Navigation, Cognitive Load, Mapping, Screen Design, Knowledge Space Com- 
plexity, Information Presentation, Media Integration, and Overall Functionality. These were elab- 
orated from the user-centered dimensions of Beeves (1992) and Reeves and Harmon (1993). For 
each of these categories, a number of statements were developed, and students were asked to 
respond to each statement in terms of an agreement rating using a five point Likert scale from 
5=strongly agree to l=strongly disagree. To establish how important each dimension was m the 
evaluation process, students were also asked to respond to each statement using an importance 
rating with a four point Likert scale from 4=very important to l=not important at all. A range ol 
statements for each of the categories was developed, for example, for Navigation: I get lost when 

searching "Encarta”; I always know where I am in Ericarta". In addition, some free generation 
questions were included that sought to clarify and elaborate on the responses students made. 

These questions were: 

What I like most about Encarta is... 

What I like least about Encarta is... 

The problems I have when using Encarta are... 

What are the advantages of using electronic information resources rather than paper 
What are the disadvantages of using electronic information resources rather than paper 

resources? 
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Table 2 shows the overall reaction to Encarta, based on composite means for each individual 
statement. 



OVERALL AGREEMENT RATING RANK ORDER 

5=Strongly Agree 1= Strongly Disagree 


Dimension 


Mearn Score 


Information presentation 


4.14 


Knowledge space complexity 


4.08 


Screen design 


3.98 | 


Media integration 


3.89 


Overall functionality 


3.81 


Ease of use 


3.68 


Mapping 


3.66 


Cognitive load 


3.49 


Navigation 


3.23 



Table 2 

At a general level, Encarta fares positively in terms of each dimension examined. Table 3 
identifies the specific features of Encarta that were ranked the highest, and lowest. 



SPECIFIC DIMENTSIONS: HIGHEST AND LOWEST AGREEMENT RATING 

5= Strongly Agree l=Strongly Disagree 


Features of Encarta students ranked highest 


Dimension 


Mean Score 


Information is presented in ways easy to understand 


4.17 


Photos and drawings are of good quality 


4.14 


Pm comfortable with using Encarta 


4.13 


Pm able to put information to good use 


4.11 


I am able to move through the contents successfully 


4.10 


Information is organized in a clear logical way 


4.09 


Information in Encarta isn’t to difficult to use 


4.09 


Features of Encarta students ranked lowest j 


Able to keep track of all the choices made while searching 


3.54 


I am able to backtrack easily 


3.51 


I always know where I am when searching Encarta 


3.63 


I understand the way all the information is organized 


3.70 


Encarta generally meets my information needs 


3.74 



Table 3 
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A different picture emertes when one explores the imortance given to each of the categories 
used for evaluation, as shown in tabesl 4 and 5. 



OVERALL IMPORTANCE RATING: RANK ORDER ~j 

4=Very Important l=Not Important at All 


Dimension 


MEAN SCORE 


Knowledge space complexity 


3.42 


Ease of use 


3.34 


Information presentation 


3.26 


Mapping 


3.24 


Media presentation 


3.20 


Overall functionality 


3.19 


Cognitive load 


3.11 


Navigation 


3.03 


Screen design 


2.97 



Table 4 



COMPARATIVE RANKINGS: AGREEMENT VS IMPORTANCE 


DIMENSION 


MEAN IMPORTANCE 


MEAN AGREEMENT 


Knowledge space complexity 


1 


2 


Ease of use 


2 


6 


Information presentations 


3 


1 


Mapping 


4 


7 


Media integration 


5 


4 


Overall functionality 


6 


5 


Cognitive load 


7 


8 


Navigation 


8 


9 


Screen design 


9 


3 



Table 5 



When searching a multimedia resource, students appear to value most the knowledge space 
complexity dimension, that is, the capacity of the package to help them make sense of new 
information in relation to what they already know; to enable them to build on what is known m 
order to complete the learning tasks required of them. The descriptive comments indicate 
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students' recognition of information as an essential component of their learning, not existing in 
isolation but rather to be integrated into their personal knowledge base. As shown in Table 5, 
students concur that Encarta fulfills their expectations with regard to the knowledge space 
complexity dimension. They agree with the publisher that they can learn from such a package. 
While students recognize ease of use as an important dimension (Rank 2) it was given a relatively 
lower ranking (Rank 6) in terms of agreement. 

Students indicate they want more from Encarta in terms of the mapping dimension. 
Strongly expressed in their co mm ents was the need to keep track of where they are at, being able 
to backtrack, or move in divergent paths and to see clearly the character of their multidirectional 
searching. As these functions are available in the package it begs the question, are they obvious 
or easily accessed by users. The package does allow for divergent search paths based on 
Categories. Outline and See indexes and continuously records each user's path through the 
electronic maze such that backward and forward re-searches are possible. The fact that all levels 
of students appear to be unaware of these functions may indicate that the screen display is either 
too crowded with options or that the function labels are not self-explanatory. 

Some beginners identified a range of attributes that primarily centered around quick and 
easy access to information, that is where the major points of information about a topic may take 
much longer to find in print resources, they are grouped together and easier to access in the 
package. In addition, they liked the pictures and film clips, with some indication that they helped 
them understand the topics. 

Average and expert users also identified the above strengths. Specific reference was made 
to language being easy to understand; information was up to date; instructions were easy to 
follow; the integration of the different media, particularly those with action, made it easier to 
understand the information, and to imagine processes and ideas. 

Obviously not all students were satisfied with various aspects of the package. 46% of 
students, at all year and experience levels represented commented on the limited quantity of 
information. They were dissatisfied with the range of topics available and the depth of coverage 
within a specific topic. While some saw it as a one- stop shop for all information requirements, 
there was also some recognition of the need to use additional sources as well to complete research 
tasks. 12% of the students expressed difficulty managing the search process utilized by the 
package and these difficulties included keeping track of position, understanding how to retrace 
steps or move on and defining search terms with managing the search process. As examples 
students referred to "information not being in a category you think it is". "I can't find the 
information I want but it's there under a different name"; and understanding how to retrace steps 
or to move on. 

Students saw a range of advantages and disadvantages in the use of electronic media 
versus print. 65% of the students said the information search was easier, more efficient and 
quicker to access, and that the information was always available with no problems incurred by 
someone else having borrowed the sources. 11% of students indicated that the task was more fun, 
more interesting to read and learn, more exciting and more enjoyable than when using print 
resources. "It doesn't seem like you are doing something for school" was a typical response. 
Several students highlighted the place of information technology in society, for example: 
"Computers are what life's all about now" and "It's our future, our jobs will use it". Average and 
expert users also identified these strengths. Specifically that the language was easy to 
understand; information was perceived as being current; instructions were easy to follow; and that 
the integration of the different media, particularly those with action, made it easier to understand 
the ideas and information. These types of claims are often made by producers and, at least in 
this case, they were upheld by the user evaluations. 

Encouraging also was the students recognition that there are disadvantages. The major 
disadvantage cited by students related to the lack of depth and complexity of information in the 
package, and this was linked to the perception that books provide greater detail and depth 
comparatively speaking. Many students expected more of the electronic resource, appearing to 
realize the capacity of the format to provide more, and they were disappointed. The confusion over 
topic descriptors and the lack of depth in some topics stems from the fact that Encarta projects the 
cultural and historical bias of the manufacturer, with very little material relevant to Australian 
history or current affairs. This is significant for students of a curriculum with emphasis on 
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national as well as international issues. The Northern American bias of the product has been 
commented on by the users in terms of its lack of information on current topics whose focus is not 



One student identified the product's potential for encouraging plagiarism in the following 
ways - "if you want the information you could print it out and just hand it in as your assignment , 
another indicated "you tend to grab too much information and not use your own brains . These 
comments typify concerns expressed by both teachers and students and emphasize the need to 
construct tasks which require more than the simple reiteration of facts. 

Another major disadvantage, though not related to Encarta per se, was the problem of 
access being affected by system downtimes. A lightning strike had caused severe damage to the 
entire network earlier in the academic year and system reconstruction took many weeks. At the 
time of the study the students had been without access for almost a term and therefore were very 
aware of the down side of information technology dependence. One student expressed this quite 
succinctly: "It's less reliable than paper, especially with system downs". Beginning users also 
identified a problem information loss; that is material either failing to print or working documents 
being deleted. These are usually rare occurrences that due to system restructuring had been 

happening quite frequently during the days preceding the study. 

It is of interest to note that some of the features of Encarta found by this study to be 
"unfriendly" or difficult to apply have been modified or removed from the 1995 edition of the 
product. For example the point at which you identify on what medium you wish to search has 
been moved forward in the structure and the procedure for copying text or image to disk or 
document has been simplified by the removal of three steps of the process. The Outline and See 
cross indexes have been altered to become a single button option titled Related Articles, which is 
plain language for the function. Other features of the 1995 version require learners to re-toarn 
where common functions can be found and new functions have been added that are, again, difficult 
to decipher. A very popular feature of the earlier editions was a search path that presented the 
image of a shelf of encyclopedias where users merely indicated the letter of the alphabet in which 
they were interested. This has been removed from the new edition. Being so similar to users 
paper based encyclopedic search methods, it was their preferred method with the electronics 
medium and its absence has caused quite a stir among many of the students. It is dear that 
multimedia producers need greater input from their prospective audiences to determine whether 
programming and glossy improvements produce, in practice, a better information product. 

The evaluation highlights the essential role of information literacy education for 
information seekers in electronic information environments. It is the dearest message of the 
findings Students' difficulties in understanding directions, commands and terminology, 
generating search terms and dealing with the mismatch of perceptions of how information is 
organized and how it actually is organized, understanding how to move in multiple directions in 
the package understanding how the information is structured and how this shapes the design of 
the search, locating related and specific information through broadening and narrowing of 
searches, dealing with the problems of little or no useful information are all issues that can be 
resolved through effective information literacy education. Possible strategies (based on Neuman 
1993) might include: individualized, hands-on instruction in searching; collaborative 

teacher-librarian and student searching; peer tutoring where expert students work with novices; 
more teacher and teacher-librarian collaboration on planning and conducting instruction that 
involves extensive electronic searching, and collaboration helping students judge the usefulness of 
electronic information retrieved. Also important is ensuring that research tasks are designed in 
such a way to develop information and reasoning skills such as comparison, contrast, analysis, 
synthesis and metacognitive abilities such as assessment, discrimination, classification, and 
judgement of information. Without these skills students will capitalize on multimedia's ease of 
use to indulge their plagiaristic habits. 

At a broader level, teacher-hbrarians need to recognize and ensure that one ol the goals ol 
incorporating electronic information sources into the curriculum is to help students master the 
higher order thinking skills involved in designing, conducting and interpreting research. Since 
teacher-librarians have an important role to play in the school in familiarizing students and 
teachers with electronics databases, their use and possibilities, they should be proficient in their 
required skills. 
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Teacher- librarians need to give careful thought to the selection of appropriate multimedia 
and to actively contribute to the multimedia debate. It is important that informed judgements are 
made about the appropriateness and limitations of multimedia and that we share this with the 
professional community; that we build an understanding of multimedia s role in meeting 
information needs, how it shapes information seeking behavior, and how it is being used, and 
what are the most appropriate support mechanisms for fostering effective use of multimedia 
packages in our agencies. 
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